Radio Resource Allocation for 5G Network Using Deep Reinforcement Learning

Abstract

Resource allocation is a critical task in 5G networks that determines how network resources are assigned to different devices and services. Traditional methods rely on predefined rules or heuristics, which may not always be optimal. Deep reinforcement learning (DRL) is a promising approach for radio resource allocation in 5G networks as it can learn to optimize resource allocation based on feedback from the network. In DRL, an agent learns to make decisions based on rewards and penalties received from the environment. In radio resource allocation, the agent would learn to allocate resources, such as frequency bands and power levels, to different devices and services to maximize some performance metric, such as throughput or energy efficiency. The main challenge in applying DRL to radio resource allocation is designing an appropriate reward function that incentivizes the agent to improve the performance metric while avoiding undesirable behavior. Additionally, the radio resource allocation problem is complex, requiring the agent to consider many variables and constraints, such as channel conditions, interference, and QoS requirements. To address this, researchers have proposed various techniques such as hierarchical RL, multi-agent RL, and curriculum learning. Despite the challenges, DRL has shown promising results in radio resource allocation for 5G networks. It has outperformed traditional methods in some scenarios, especially when network conditions are dynamic and unpredictable. However, further research is necessary to explore the scalability and robustness of DRL-based approaches in practical 5G networks. In this method we suggest an algorithm for voice and data carriers in sub-6 GHz and millimeter wave (mmWave) frequencies respectively. The mmWave ranges between 30GHz to 300GHz.

Introduction

In wireless networks, Radio Access Networks (RANs) are becoming increasingly sophisticated and complex as they become a component of  the LBSs  they blend, the explosion of intelligent routing products as well as the service s and features they support that are disruptive to the industry. With this advent of quinary generation wireless technology (5G), the gigantic growth of business capacity and rate of data continues. With a better codec and better responsibilities, voice calls are also improving. A RL frame is a basis for our proposed online knowledge-based algorithm.

In this article, we examine the most widely used DRL methods to address RRAM complication, including merit and plan based methods. There is a discussion of the improvement, restraint, and applications of each method. The literature analysis includes twain addressing of radio resources and probing mobile networks, and we categorize being combined factory accordingly. Therefore, we describe precisely the variety of DRL methods used in each allied work, their rudiments, and the main conclusions of all allied work. In the end, we highlight significant unresolved issues and provide insight into a number of potential future discourse routes in the context of deep study- rested radio source handling. It's touted as one of the coming big digital revolutions and because of how important it's to maintain connectivity to critical bias controlling our safety and security, improvement will be demanded in the network response times or quiescence. Background with the advent of quinary generation wireless shipping, the rate of business capacity and info growth is continuing to increase ( 5G). Improved voice calling with higher responsibility and stronger codecs is another development. So, it is projected that mobile networks will be capable of handling this immediate requirement for both the increased speech quality as well as data . We introduce an active experience and understanding approach that is based on a strengthening knowledge frame to understand the inferred properties of cellular terminal. In order to improve the end-user signal to interference plus noise and sum-estimate ability, we use this frame to determine a nearly optimal strategy. (1) provides evidence of the importance of improving power management expertise (3). Voice communications are strengthened against wireless impairments, such as fading, thanks to power management . Moreover, it increases cellular capacity and improves network usability. Shaft formation, power management, and interference collaboration for data connections    can    improve    the    reliability     of     such info connections, improve the data rates entered by customers, avoid forwarding. ii. Summary of current research Massive MIMO common power dominance was launched in ( 4). Because to the limited transmission of channel   case details between the base station taking part in the popular power dominance, this technique resulted in a reduced above. The SINR showed improved performance as measured by the common power dominance method. Power dominance with shaft formation was investigated in the uplink supervision ( 5). An optimization problem was created to raise the maximum estimate rate for the both users while preserving a minimal cost constraint for each user.

The user equipment (UE) battery may drain more quickly if the challenge for the uplink is solved via reinforcement learning, which is computationally intensive. On the other hand, we emphasise power control, beam formation, interference cancellation, and the downlink.

A. Problem Statement

By presenting an alternative way for power control in wireless grid, we respond to the query of whether there is a technique that will execute the combined BF,PC, and IC. As in normal implementations, in such a situation the transmit capacity of the helping BS is regulated, but also, as illustrated in Figure 1, transmit power to obstructing base stations from a chief location.

Thus, a key question is if there is a process that

1. Capable of solving power control, interference coordination, and beam shaping problems cooperatively
2. Reach the SINR upper bound, and
3. Forgoes the time-consuming action space search for both bearer kinds.

In order to traverse the solution space by studying from interaction, reinforcement learning is used in this research to offer method for this joint result.

This technique is used for voice and data carriers. In addition, we investigate the overhead brought about by sending data to a central site that uses online learning to compute the answer.

B. Objectives

The aim of maximizing the succeeding discounted benefit for all action pick by the representative, which results in the environment swapping circumstances, governs the interaction of these elements.

The policy controls how the agent and the state interact with one another. We learn the value of the anticipated reduced reward throughout the training period. The beam forming vectors and transmit powers at the base stations are concurrently managed in our proposed DRL-based algorithm to optimise the goal function in (6). We can execute multiple actions simultaneously by using a group of bits as an activity register. By choosing samples of experience at random, we do minibatch coaching the DQN. The advantages of this perspective are stability and the avoidance of local minimum convergence. The use of incident replay also supports the application of off-policy studying approaches because this present DQN framework differ from those that were used to bring the sample from D.

II. LITERATURE SURVEY

In Check In(1), a communications pathway voice over LTE (VoLTE) radio deliverer method for an inner terrain served by little cells is proposed. This method is RL-grounded and unbounded circular powered. The two main contributions of this paper are to 1) use RL to solve performance tuning issues in an inner cellular grid for call liaisons and 2) demonstrate that the deduced lower set loss in effective signal to hindrance plus noise rate due to bordering cell failure is adequate for VoLTE power dominance in cellular grid. When compared to present assiduity standards, the simulation shows a considerable improvement in both voice retention capabilities and mean thought score from the suggested RL- grounded power regulation algorithm In this analysis (2), they investigated the potential interactions between non-orthogonal couple access and videlicet milli metre wave (mm-surge) dispatches,   two   key   enabling    automation for    the quinary generation of wi-fi cell communication. They model a typical two-stoner uplink mm-surge-NOMA system in which the base stations supplies two NOMA drug addicts with food and equips a structure that produces analogue light with a single radio-frequency chain. An optimization problem is created in order to maximise the possible aggregate rate of the two drug users while adhering to a minimal charge limit for each user. The issue becomes one of common power management and ray formation, requiring us to simultaneously control the appropriate capacity on the two drug users while determining the ray forming vectors to drive to them. Extensive reconstruction support the rationale of the proposed method, and the execution assessment results demonstrate that the proposed sub-capital result provides a close-to-vault uplink sum rate effectiveness. The existing wireless grid is developing into a 5G wireless communication system that uses several technology to increase its structure capacity in order to handle the data(3). Since 5G wireless communication systems use mmWave transmissions, we note that NLOS transmission is pervasive in mobile communication and is really more widespread. Former workshops use ray forming techniques to improve NLOS transmission execution, but they struggle with the expensive antenna management costs. For optimising NLOS performance of transmission, we suggest a dynamic transmission power control technique in this study. This research proposes a deep Q- network (DQN) strategy, in which we use a convolutional neural network, to solve the maximisation problem. It provides simulation data to demonstrate the effectiveness of the suggested plan. Base stations and mobile stoner outfits (UEs) undertake ray forming operations in paper(4) various ultra-compact networks with millimetre large-scale cells and small scale cells to establish mostly directed links. As more BSs are densely stationed, the malignancy of the spatial diversity attain through directional links increases, causing significant inter-cell interference from synchronous directional transmissions of conterminous BSs, which results in a decline in downlink performance in the network. Yet, due to nature of the time-differing wireless terrain, the vigorous change in ray broadcasting direction, and the moveable device position, inter-cell obstruction is extremely difficult to handle. It suggests an online literacy- based communication and cooperation method built on this multigame structure. We confirm the efficiency of the suggested online literacy- grounded inter-BS hindrance operation system using numerical simulations. Article (5) examines a dynamic multichannel access problem in which drug users choose the channel to transmit data over numerous corresponding channels that follow an unidentified joint Markov model. Finding a strategy which optimises the anticipated number of fruitful transmissions over the long term is optimal. The issue is described as a Markov decision process with an unidentified system dynamic that is only partially observable. We confirm the efficiency of the suggested online literacy- grounded inter-BS hindrance operation system using numerical simulations. In Paper (5), a dynamic multichannel access problem is examined in which drug users choose the channel to broadcast over numerous correlated channels that follow an unidentified joint Markov model. In order to demonstrate that DQN may attain the same optimal performance without being aware of the system statistics, they first investigate the best policy for fixed pattern channel switching under the assumptions of the system dynamics. It suggests a DQN strategy that is flexible and has the capacity to change its script literacy over time. In(6), Automatic modulation recognition is a demanding and difficult content in the evolution of the cognitive radio, and it is a basis of robust modulation and demodulation competencies that feel the study surroundings and make matching adaptations. AMR is fundamentally a bracket problem, and deep literacy attains high-quality execution in colorful bracket tasks. This paper offers a deep literacy- grounded system, mixed with the two convolutional neural networks (CNNs) instruct on specific datasets, to attain superior delicacy AMR. The CNN is instructed on samples collected in- section and quadrature component signals, else recognized as in- section and quadrature samples, to differentiate modulation modes which are convenient to recognize. It borrow powerhouse as an alternative of pooling operation to attain advanced consciousness delicacy. A CNN built using constellation plates is likewise intended to showcase difficult-to-distinguish modulation types. In this paper(7), many bumps working out the linked introduction simultaneously broadcast data across the same time-frequency buckets, providing profound proficiency(DL) grounded on-orthogonal subjective get to(NORA). Effective manipulation control calculations are essential for the NORA even when there are only partial records of the channels available due to the timing development (TA).

This poses difficulties for calculations that use all available channel information. It implies unmonitored DL- grounded control manage plans which maximize the negligible fee grounded as it have been on the TA data. Numerical comes about show the viability of the proposed DL based NORA over typical styles. In organize to enhance flag substance of mmWave systems, prepare densification should be utilized at the same time. Paper(8) A profound education- grounded beam operation and prevention collaboration( BM- IC) framework in thick mmWave organize is proposed to tackle the drawback Simulation comes about appear that the proposed profound proficiency- grounded BM- IC approach can select up comparative whole- price to customary BM- IC calculation however with tons decrease calculation time. Due to serious flag pathloss in millimeter surge(mmWave) band, beamforming empowered directional transmission is basic to overcome the weakening challenge in unborn mmWave verbal exchange frameworks. Paper [9] it employments sub6 Ghz in FDM for heterogenous base stations which increments downlink patron throughput conveyance. This method Resolve the race circumstance between the base stations in sub- exponential times inside the number of radio wires which employments surrogate comp set off work. The race condition is managed through a central location based upon the consumer specific downlink SINR and arranges. In [10] The proposed calculation in contrast to wellknown handover is probable to fall flat or succeed Enhancement in handover victory charges which assembles ample data to Anticipate handover. Made strides Inter-RAT handover victory will keep the classes in ideal band for longer time on victory rate. The quantity of disappointments in both calculations is comparative in this strategy with the slightest quantity of purchasers and most limited term. In [11] We discovered the upward thrust to pick out up transmission diagram that can finish the fundamental remarkable SINR of a nonlinear MISO device because it had been at the point of neighborhood ideal transmit manage via examination of the flag to obstructions also commotion extent (SINR). After we decided an evaluated perfect transmit control for the nonlinear MISO system. We proposed a precoding and manage controlling calculation for a MISO machine with nonlinear manage audio system underneath control confinement to will increase the SINR in all transmit manipulate region.In [12] Working at millimeter Wave recurrence band and large scale base stations (MBSs) working at sub6 Giga Hertz coexist utilized to examine the execution of 5G communication systems where the SBSs with facilitated pillar shaping. A clustering strategy is proposed to select a few SBSs to dispense with intra-cell obstructions. After, to get signal to interference ratio (SINR) and rate scope likelihood expressions we have put an normal remove from the Kth SBS to a client. Another, MBSs work at sub-6 GHz and SBSs work facilitated beamforming with 28 GHz in 5G mmWave organize is dissected.

In [13] we overcome this problem, and each user gear (UE) is only geared up with one single antenna. As a result, the combination of beam-forming with a exclusive modulation scheme for the DL is proposed which is improved by frequency diversity. The frequency range is additionally incorporated to allow the integration of the exclusive modulation schemes in the DL, the place UEs are restricted to equip with a small quantity of antennas (single antenna is used in this work).

In [14] an synthetic neural network that is educated the use of the training statistics generated by using solving the offline power control hassle is trained. Simulations indicate that the proposed approach presents very good performance.

[15] discusses how blockage radically affects insurance and reliability of highly-mobile links the use of slim beams and the sensitivity of mmWave signals. The proposed approach is evaluated using simulations with realistic channel models and mobility patterns. The results show that the deep learning-based approach outperforms traditional beamforming techniques in terms of throughput and energy efficiency, particularly in highly-mobile scenarios.

In [16] In order to improve the reliability to give up users, tuning the cellular network in opposition to wireless impairment. To furnish a answer to enhance the performance in indoor and outdoor environments, we formulated cell community overall performance tuning as a reinforcement mastering (RL) trouble in this paper. The authors also propose a hierarchical approach to RL, where the agent learns at different levels of abstraction to optimize different network objectives. The highest level of abstraction optimizes the overall network performance, while lower levels optimize specific network parameters such as power allocation and handover thresholds. The proposed framework is evaluated using simulations with realistic network scenarios and performance metrics. The results show that the RL-based approach outperforms traditional methods in terms of network performance and scalability, particularly in dynamic and unpredictable environments. Overall, the paper presents a novel framework for automating the tuning of cellular networks using RL, which addresses the challenges of manual tuning and rule- based heuristics in complex and large-scale networks. The proposed framework shows promising results and highlights the potential of RL for future network optimization. Regenerate response A beam forming method is described in [17] for enhancing the signal first-class in multiuser multiple-input-single-output systems. Conventionally, finding the superior beam forming infusion relies upon on continual method which end result in computing prolong and are not appropriate for real time execution.

It [18] introduces a novel algorithm for optimizing the performance of massive MIMO beamforming. The combination of three neural networks is the key innovation of the algorithm which cooperatively implements the deep destructive reinforcement getting to know workflow.

III. CONSOLIDATED TABLE

IV. ACKNOWLEDGEMENT

Every accomplishment not only depend on the individual work but on the advice, motivation and association of intellectuals, teachers and friends. We advance our wholehearted thanks to Dr. Kamalakshi Naganna, Professor and Head, Department of Computer Science and Engineering, Sapthagiri College of Engineering, and Dr. PraveenKumar K V, Professor, Department of Computer Science and Engineering, Sapthagiri College of Engineering, for consistent help, advice, and regular assistance throughout work. Finally, we thank our parents and friends for their moral support.

Conclusion

This paper says, the solution for beam forming, power control, and interference coordination as a non-convex optimization joint outline to enhance the signal to interference plus noise ratio (SINR) using deep reinforcement learning. This paper gives following step in giving a different system to control power in wireless-networks: The joint beam forming, powercontrol, and interference-coordination problem is formulated in the downlink direction as an development issue that improves the user’s received SINR. The race condition between the base stations in sub-exponential times is resolved by the wide variety of antennas. In order to manipulate the race condition, a central vicinity is required to file downlink SINR and coordinates as stated by using the user. This paper suggests how to create a solution based totally on reinforcement learning. In this solution, extra than one motion is taken at the equal time through binary encoding of the associated moves which is carried out by way of the BS. As an choice to beam-forming, the above approach was once used to improve the performance of non-line of sight (NLOS) transmissions. Using deep reinforcement learning, the problem of allocation of power to increase the sum rate of UE below the manage of transmission electricity and best used to be solved. A convolutional NN is the one used for the estimation of the Q function of the deep reinforcement getting to know hassle in the above solution.

References

[1] F. B. Mismar and B. L. Evans, “Q-Learning Algorithm for VoLTE Closed Loop Power Control in Indoor Small Cells,” in Proc. Asilomar Conference on Signals, Systems, and Computers, Oct. 2018. [2] L. Zhu, J. Zhang, Z. Xiao, X. Cao, D. O. Wu, and X. Xia, “Joint Power Control and Beamforming for Uplink Non- Orthogonal Multiple Access in 5G Millimeter-Wave Communications,” IEEE Transactions on Wireless Communications, vol. 17, no. 9, pp. 6177–6189, Sep. 2018. [3] C. Luo, J. Ji, Q. Wang, L. Yu, and P. Li, “Online Power Control for 5G Wireless Communications: A Deep Q- Network Approach,” in Proc. IEEE International Conference on Communications, May 2018. [4] R. Kim, Y. Kim, N. Y. Yu, S. Kim, and H. Lim, “Online Learning-based Downlink Transmission Coordination in Ultra-Dense Millimeter Wave Heterogeneous Networks,” IEEE Transactions on Wireless Communications, vol. 18, no. 4, pp. 2200–2214, Mar. 2019. [5] Wang, H. Liu, P. H. Gomes, and B. Krishnamachari, “Deep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networks,” IEEE Transactions on Cognitive Communications and Networking, vol. 4, no. 2, pp. 257–265, Jun. 2018. [6] Y. Wang, M. Liu, J. Yang, and G. Gui, “Data-Driven Deep Learning for Automatic Modulation Recognition in Cognitive Radios,” IEEE Transactions on Vehicular Technology, vol. 68, no. 4, pp. 4074–4077, Apr. 2019. [7] S. Jang, H. Lee, and T. Q. S. Quek, “Deep learning-based power control for non-orthogonal random access,” IEEE Communications Letters, pp. 1–1, Aug. 2019. [8] Pei Zhou, Xuming Fang, Senior Member, IEE, Xianbin Wang, Fellow, IEEE, Yang Long, Member, IEEE, Rong He, and Xiao Han “Deep Learning Based Beam Management and Interference Coordination in Dense mmWave Networks” IEEE Transactions on Aerospace and Electronic Systems. [9] F. B. Mismar and B. L. Evans, “Deep Learning in Downlink Coordinated Multipoint in New Radio Heterogeneous Networks,” IEEE Wireless Communications Letters, vol. 8, no. 4, pp. 1040–1043, Aug. 2019. [10] F. B. Mismar and B. L. Evans, “Partially Blind Handovers for mmWave New Radio Aided by Sub-6 GHz LTE Signaling,” in Proc. IEEE International Conference on Communications Workshops, May 2018. [11] Jeongju Jee, Girim Kwon and Hyuncheol Park “Precoding Design and Power Control for SINR Maximization of MISO System With Nonlinear Power Amplifiers” IEEE Transactions on Vehicular Technology ( Volume: 69, Issue: 11, November 2020). [12] Sisai Fang; Gaojie Chen; Xiaodong Xu; Shujun Han; Jie Tang “Millimeter-Wave Coordinated Beamforming Enabled Cooperative Network: A Stochastic Geometry Approach” IEEE Transactions on Communications 2021. [13] Kun Chen-Hu; Yong Liu; Ana García Armada “Non- Coherent Massive MIMO-OFDM Down-Link based on Differential Modulation” IEEE Transactions on Vehicular Technology 2020. [14] Mohit K. Sharma; Alessio Zappone; Mérouane Debbah; Mohamad Assaad “ Deep Learning Based Online Power Control for Large Energy Harvesting Networks ” IEEE Transactions on Cognitive Communications and Networking 2019. [15] Ahmed Alkhateeb; Sam Alex; Paul Varkey; Ying Li; Qi Qu; Djordje Tujkovic “Deep Learning Coordinated Beamforming for Highly-Mobile Millimeter Wave Systems ” IEEE Signal Processing 2019. [16] Faris B. Mismar; Jinseok Choi; Brian L. Evans “A Framework for Automated Cellular Network Tuning with Reinforcement Learning” IEEE Transactions on Communications 2019. [17] Wenchao Xia; Gan Zheng; Yongxu Zhu; Jun Zhang; Jiangzhou Wang “A Deep Learning Framework for Optimization of MISO Downlink Beamforming” IEEE Transactions on Communications 2020. [18] Taras Maksymyuk; Juraj Gazda; Oleh Yaremko; Denys Nevinskiy “ Deep Learning Based Massive MIMO Beamforming for 5G Mobile Network x” 2018 IEEE 4th International Symposium on Wireless Systems.

Copyright

Copyright © 2023 Umesh R G , Sushil Kumar G N, Santhosh K, Suraksha M S, Dr. Praveen Kumar K V. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.